A Review on Storage and Task Scheduling in Heterogeneous Hadoop Clusters
نویسندگان
چکیده
The task scheduling algorithm for homogeneous Hadoop clusters is incapable of proper utilization of resources in heterogeneous clusters. To overcome this issue, an adaptive task scheduling algorithm has been proposed. With adaptive task scheduling we aim for better resource utilization by dynamically adjusting the workload at runtime. Also we are making the storage of data resource aware so that the load is balanced according to processing capabilities. With the adaptive strategy task trackers can self-regulate the tasks while maintaining performance and providing stability and scalability. The adaptive approach promises higher efficiency and better performance.
منابع مشابه
An Optimal Task Assignment Policy and Performance Diagnosis Strategy for Heterogeneous Hadoop Cluster
The goal of the proposed research is to improve the performance of Hadoop-based software running on a heterogeneous cluster. My approach lies in the intersection of machine learning, scheduling and diagnosis. We mainly focus on heterogeneous Hadoop clusters and try to improve the performance by implementing a more efficient scheduler for this class of cluster.
متن کاملAnalysis of Information Management and Scheduling Technology in Hadoop
Development of big data computing has brought many changes to society and social life is constantly digitized. ‘How to handle vast amounts of data’ has become a more and more fashionable topic. Hadoop is a distributed computing software framework, which includes HDFS and MapReduce distributed computing method, make distributed processing huge amounts of data possible. Then job scheduler determi...
متن کاملAn Efficient Genetic Algorithm for Task Scheduling on Heterogeneous Computing Systems Based on TRIZ
An efficient assignment and scheduling of tasks is one of the key elements in effective utilization of heterogeneous multiprocessor systems. The task scheduling problem has been proven to be NP-hard is the reason why we used meta-heuristic methods for finding a suboptimal schedule. In this paper we proposed a new approach using TRIZ (specially 40 inventive principles). The basic idea of thi...
متن کاملAn Efficient Genetic Algorithm for Task Scheduling on Heterogeneous Computing Systems Based on TRIZ
An efficient assignment and scheduling of tasks is one of the key elements in effective utilization of heterogeneous multiprocessor systems. The task scheduling problem has been proven to be NP-hard is the reason why we used meta-heuristic methods for finding a suboptimal schedule. In this paper we proposed a new approach using TRIZ (specially 40 inventive principles). The basic idea of thi...
متن کاملAnalysis and Optimization of the Hadoop Speculative Execution Mechanism
The existing Hadoop clusters are mostly composed of heterogeneous nodes, which have different computing and storage capacities, with the speed of maps to reduce tasks performed on the nodes being quite different. However, the finish time of the entire job is determined by the slowest task, so looking for the “drag tasks” strategy has a dominant position in the whole job scheduling process. The ...
متن کامل